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FOREWORD 


The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  is 
examining  distributed  learning  technologies  for  use  by  soldiers  in  an  “on  demand” 
environment,  where  training  becomes  more  soldier  centered  rather  than  classroom  based. 
Part  of  the  work  involves  a  thorough  assessment  of  the  distance  learning  research 
literature  as  it  pertains  to  training.  The  present  report,  under  ARI’s  WEBTRAIN  project, 
provides  an  examination  of  empirical  evidence  concerning  the  effectiveness  of  Web- 
based  instruction.  It  forms  a  foundation  for  guidelines  to  help  training  developers, 
training  managers,  and  policy  makers. 

The  results  of  this  research  were  presented  to  the  U.S.  Army  Training  and 
Doctrine  Command,  Assistant  Deputy  Chief  of  Staff  for  Training,  on  17  October  2002. 


Acting  Technical  Director 
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THE  EFFECTIVENESS  OF  WEB-BASED  INSTRUCTION 


EXECUTIVE  SUMMARY _ _ _ _ _ 

Research  Requirement: 

The  Army  is  embarking  on  a  major  change  to  deliver  standardized  individual,  self¬ 
development,  and  small  group  training  to  soldiers  through  the  application  of  networked 
communication  systems.  Planning  calls  for  Army  training  to  become  more 
learner-centric,  in  which  soldiers  assume  increased  responsibility  for  the  acquisition  of 
knowledge  and  the  development  of  skills,  delivered  over  the  Internet  or  military  intranets. 
Reports  in  the  literature  on  the  effectiveness  of  instruction  delivered  over  the  Web  have 
begun  to  appear,  but  there  has  not  been  an  integrated  analysis  of  these  early  findings. 

This  report  offers  an  initial  analysis  of  the  reported  findings. 

Procedure: 

Searches  on  relevant  research  databases  yielded  40  reports  with  learning  outcome  data 
from  the  more  than  500  reports  on  the  topic  of  Web-based  instruction.  When  including 
multiple  studies  documented  in  some  reports,  the  number  of  cases  that  provided  empirical 
evidence  increased  to  47.  Of  these,  15  provided  sufficient  data  to  compute  an  effect  size, 
a  standard  way  of  quantifying  differences  between  groups.  Here,  the  groups  refer  to  a 
comparison  group  that  received  instruction  in  a  conventional  classroom  and  an 
experimental  group  that  received  comparable  instruction  over  the  Web. 

Findings: 

The  average  effect  size,  comparing  Web-based  instruction  to  conventional  classroom 
instruction,  was  .24,  which  means  the  “average”  student  moved  from  the  50th  to  the  59th 
percentile.  The  relatively  small  number  of  studies  reporting  outcome  data  demonstrated 
wide  variability;  the  effect  sizes  ranged  from  -.4  to  1 .6.  In  earlier  analyses  of  the 
effectiveness  of  non  Web-based  forms  of  computer-based  instruction  that  used  hundreds 
of  studies,  effects  sizes  of  between  .32  and  .41  have  been  reported  (corresponding  to  the 
63rd  and  66th  percentiles).  In  terms  of  instructional  effectiveness,  it  appears  that  current 
practices  in  Web-based  instruction  lead  to  an  improvement  in  learning  when  compared  to 
the  classroom,  but  the  central  tendency  falls  short  of  that  for  traditional  computer-based 
instruction.  In  view  of  the  shortage  of  empirical  findings  and  the  relative  novelty  of 
using  the  Web  for  instructional  purposes,  dozens  of  more  controlled  studies  are  needed 
before  a  more  accurate  effect  size  can  be  resolved.  There  is  no  practical  reason  why  the 
effect  size  should  be  less  than  that  demonstrated  for  computer-based  instruction. 

Utilization  of  Findings: 

These  findings  can  be  used  as  a  basis  for  (1)  selecting  best  practices  in  an  objective 
manner,  (2)  establishing  objective  performance  criteria  for  future  learning  content,  and 
(3)  creating  an  initial  data  set  to  which  future  evidence  on  the  effectiveness  of  Web-based 
instruction  can  be  added. 
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THE  EFFECTIVENESS  OF  WEB-BASED  INSTRUCTION 

The  U.S.  Army  plans  to  increase  the  use  of  distributed  learning  to  meet  the  future  training 
needs  of  a  responsive  and  versatile  force.  The  goals  of  the  Army  Training  and  Doctrine 
Command  are  to  provide  digital  training  facilities  within  50  miles  of  95  percent  of  Army  duty 
stations  and  develop  more  than  525  distributed  learning  courses.  The  courses  will  encompass 
online  instruction  distributed  over  communication  networks,  blended  to  varying  degrees  with 
CD-ROM,  print  material,  email  and  chats,  and  conventional  hands-on  instruction  and  testing  as 
needed.  For  the  versatile  force  of  the  future,  learning  will  be  centered  around  the  soldier  instead 
of  the  classroom.  A  central  question,  then,  is  just  how  effective  is  online  instruction?  In 
particular,  how  does  it  compare  to  both  the  conventional  classroom  and  established  forms  of 
stand-alone,  computer-based  instruction  (CBI)  prior  to  the  World  Wide  Web? 

For  the  purposes  of  this  review,  online  instruction  is  considered  to  be  any  educational  or 
training  program  distributed  over  the  Internet  or  an  intranet  and  conveyed  through  a  browser, 
such  as  Internet  Explorer™  or  Netscape  Navigator™.  Hereafter,  it  is  referred  to  as  Web-based 
instruction.  The  use  of  browsers  and  the  Internet  is  a  relatively  new  combination  in  instructional 
technology.  While  the  effectiveness  of  traditional  CBI  has  been  reviewed  quite  thoroughly 
(Kulik,  1994;  Lou,  Abrami,  &  d’Apollonia,  2001),  the  effectiveness  of  online  instmction  has 
received  little  analysis.  Part  of  the  reason  is  because  few  cases  have  been  detailed  in  the 
literature.  This  report  serves,  then,  as  an  initial  look  at  the  empirical  evidence  for  its 
instructional  effectiveness. 

Advantages  of  Training  Technology 

The  Army  seeks  to  take  advantage  of  the  benefits  offered  by  distributed  learning,  such  as 
reduced  travel  costs,  increased  accessibility,  and  improvements  in  learning.  The  cost  advantage 
of  even  basic  forms  of  technology-based  instruction  has  been  documented  (Fletcher,  1990; 
Wisher,  Priest,  &  Glover,  1997).  Learning  advantages  have  consistently  been  found  whenever 
well-designed  instruction  is  coupled  with  computer  delivery.  Fletcher  (2001),  for  example,  has 
established  the  “Rule  of  Thirds”  based  on  an  extensive  review  of  the  empirical  findings  in 
educational  and  training  technology.  This  rule  advises  that  the  use  of  CBI  reduces  the  cost  of 
instruction  by  about  one-third,  and  additionally  either  reduces  the  time  of  instruction  by  about 
one-third  or  increases  the  effectiveness  of  instruction  by  one-third.  The  analysis  for  this  rule  was 
based  primarily  on  stand-alone  CBI,  not  the  contemporary  use  of  online  technologies. 

There  is  no  practical  reason  to  expect  Web-based  instruction  to  be  any  less  effective  than 
traditional  CBI.  Both  are  capable  of  interactivity,  individual  feedback,  and  multi-media 
presentation.  However,  technical  limitations  with  current  Web-based  configurations  may  dilute 
some  of  these  advantages.  Inherent  limitations  such  as  a  small  viewing  area  for  video,  video 
with  a  slow  frame  speed,  or  delays  in  responsiveness  as  a  result  of  high  traffic  load  on  the 
Internet  may  restrict  its  current  effectiveness.  On  the  other  hand,  Web-based  instmction  offers 
new  advantages  to  the  learner,  such  as  interactivity  with  instructors  and  students  and  quick 
access  to  supplementary  online  resources.  As  the  technology  improves,  Web-based  instmction 
may  have  an  ultimate  advantage. 
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This  report  first  reviews  the  numerous  ways  in  which  the  World  Wide  Web  is  influencing 
instructional  practices.  Then,  the  parameters  for  and  results  of  a  search  on  the  evaluation 
literature  are  described.  The  results  of  a  meta-analysis  of  the  evaluation  data  are  reported  and  a 
central  tendency  of  instructional  effectiveness  is  identified.  How  well  this  central  tendency, 
represented  as  a  standardized  effect  size,  compares  to  both  the  conventional  classroom  and  to 
traditional  uses  of  the  computer  as  a  stand-alone  delivery  device  is  discussed.  Finally,  some 
conclusions  are  drawn  from  this  initial  inquiry  into  the  effectiveness  of  Web-based  instruction. 

The  Use  of  the  World  Wide  Web  In  Course  Instruction 

The  World  Wide  Web  can  be  used  in  various  ways  for  instructional  and  instructional 
support  purposes.  For  example,  Web-based  instruction  offers  the  learner  unparalleled  access  to 
instructional  resources,  far  surpassing  the  reach  of  the  conventional  classroom.  Web-based 
instruction  also  makes  possible  learning  experiences  that  are  open,  flexible,  and  distributed, 
providing  opportunities  for  engaging,  interactive,  and  efficient  instruction  (Kahn,  2001).  Phrases 
such  as  “flexible  navigation,”  “richer  context,”  “learner  centered,”  and  “social  context  of 
learning,”  are  populating  the  literature  on  Web-based  instruction.  Furthermore,  the  rendering  of 
cognitive-based  theories  of  learning  into  the  terminology  of  Web-based  instruction  extends  the 
technical  nomenclature  to  actual  instructional  practices  (Bonk  &  Dennen,  1999).  Indeed,  Dills 
and  Romiszowksi  (1997)  have  identified  more  than  40  instructional  paradigms  seeking  to 
advance  and  improve  the  online  learning  experience  beyond  the  conventional  classroom. 

Some  researchers  have  argued,  however,  that  the  tried-and-true  principles  of  instructional 
design,  namely  interaction  and  timely  feedback,  are  often  absent  from  Web-based  instruction, 
particularly  from  individual  websites  devised  to  teach  (Eli-Tigi  &  Branch,  1997).  The  absence 
of  a  sturdy  pedagogical  underpinning  for  a  Web-based  program  can  diminish  an  otherwise 
worthy  opportunity  to  improve  learning.  Well-designed  CBI  developed  in  the  1970s  and  1980s, 
for  example,  has  demonstrated  an  enhancement  to  learning  outcomes  when  compared  to 
classroom  instruction  (Kulik  &  Kulik,  1991).  How  does  Web-based  instruction  compare  to  the 
classroom  at  this  relatively  early  stage  of  its  implementation?  In  view  of  the  extraordinary 
potential  for  Web-based  instruction,  it  is  somewhat  surprising  that  there  are  few  empirical 
assessments  of  its  effectiveness. 

Unlike  the  fixed  resources  in  conventional  GBI,  Web-based  instruction  can  be 
conveniently  modified  and  redistributed,  readily  accessed,  and  quickly  linked  to  related  sources 
of  knowledge.  This  establishes  a  backbone  for  “anytime,  anywhere”  learning  (Fletcher  & 
Dodds,  2001).  Compare  these  features  to,  say,  a  pre-Internet  CD-ROM  in  which  instructional 
messages  were  encoded  in  final  form,  availability  was  limited  to  specific  computers,  and 
immediate  access  to  a  vast  array  of  related  materials  was  not  possible.  To  be  sure,  many  key 
instructional  features,  such  as  learner  control  and  feedback,  are  shared  between  Web-based  and 
conventional  CBI. 

A  reasonable  assumption  concerning  the  effectiveness  of  Web-based  instruction,  then,  is 
that  it  should  be  at  least  “as  good  as”  conventional  forms  of  CBI.  That  is,  the  effect  size  of 
evaluations  that  compare  the  effectiveness  of  either  to  the  classroom  should  be  comparable.  The 
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qualities  shared  by  the  two  delivery  media  include  multi-media  formats,  self-pacing,  tailored 
feedback,  and  course  management  functions.  Additionally,  the  unique  features  of  Web-based 
instruction  -  flexible  courseware  modification,  broad  accessibility,  and  online  links  to  related 
materials,  instructors,  and  fellow  students  -  should  make  possible  improvements  in  learning 
outcomes  beyond  what  CBI  has  demonstrated.  Within  the  CBI  literature  as  described  in  detail 
shortly,  the  learning  outcomes  (when  compared  to  conventional  classroom  instruction)  have 
demonstrated  effect  sizes  of  between  0.3  and  0.4  standard  units  (Fletcher,  1990,  Kulik,  1994). 

The  measure  of  effect  size  is  simply  a  way  of  quantifying  the  difference  between  two  groups. 
For  example,  if  one  group  had  an  experimental  treatment  (Web-based  instruction)  and  the  other 
did  not  (the  conventional  classroom),  then  the  effect  size  is  a  measure  of  the  effectiveness  of  the 
Web-based  treatment  compared  to  that  of  the  classroom.  In  statistical  terms,  the  effect  size 
describes  the  difference  between  two  group  means  divided  by  either  the  pooled  standard 
deviation  or  the  standard  deviation  of  the  treatment  group  (Glass,  McGaw,  &  Smith,  1991).  An 
advantage  of  using  effect  size  is  that  numerous  studies  can  be  combined  to  determine  an  overall 
best  estimate,  or  central  tendency,  of  the  effect.  Generally,  values  of  0.2, 0.5,  and  0.8  standard 
units  are  considered  to  correspond  to  small,  medium,  and  large  effect  sizes,  respectively  (Cohen, 
1988).  (N.B.  Effect  sizes  can  be  quickly  translated  to  percentiles  from  a  table  of  standard 
distributions.)  Our  premise  in  the  present  analysis  is  that  if  Web-based  instruction  is  employed 
properly,  and  in  a  similar  manner  to  conventional  CBI,  it  should  lead  to  an  effect  size  that  is 
comparable  to  that  demonstrated  for  CBI.  The  assumption  of  being  “as  good  as”  CBI  is 
reasonable,  of  course,  when  the  Web  is  applied  for  purposes  similar  to  conventional  CBI,  namely 
for  achieving  specific  learning  objectives. 

Roles  of  the  Web  In  Instructional  Settings 

Web-based  instruction  presents  multiple  dimensions  of  use  in  education  and  training 
environments.  As  with  CBI,  it  is  capable  of  providing  direct  instruction  to  meet  individual 
learning  objectives.  Due  to  its  networking  capability,  the  Web  can  assume  additional  roles. 

These  include  promoting  and  facilitating  enrollment  into  courses,  availing  the  syllabus  or 
program  of  instruction,  posting  and  submitting  assignments,  email  between  instructors  and 
fellow  students,  collaboration  on  assignments,  and  building  learning  communities. 

The  Web  has  become  a  powerful  tool  for  learning  and  teaching  at  a  distance.  Its  inherent 
flexibility  allows  application  in  a  variety  of  ways  within  an  educational  context,  ranging  from 
simple  course  administration  and  student  management  to  teaching  entire  courses  online.  Each  of 
these  “levels  of  use”  works  towards  a  different  goal.  These  goals  should  be  recognized  when 
evaluating  the  use  of  the  Web.  For  example,  an  instructor  may  hold  face-to-face  lectures  in  a 
classroom  but  post  the  class  syllabus,  assignments,  and  grades  on  the  Web.  In  this  case,  it  would 
not  be  appropriate  to  evaluate  the  use  of  the  Web  with  respect  to  learning  outcomes,  since  the 
Web  was  not  used  in  a  direct  instructional  role. 

There  are  a  host  of  factors  that  contribute  to  a  meaningful  learning  environment.  In  an 
attempt  to  gain  a  systematic  understanding  of  these  factors,  Kahn  (1997)  developed  a  framework 
for  Web-based  learning,  consisting  of  eight  dimensions:  pedagogical,  technological,  interface 
design,  evaluation,  management,  resource  support,  ethical,  and  institutional.  Kahn  (2001)  later 
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offered  a  framework  for  the  extent  of  Web-based  instruction  along  a  continuum  ranging  from 
“micro”  uses  to  “macro”  uses.  The  “micro”  end  of  the  continuum  involves  the  use  of  the  Web  as 
a  way  to  supplement  or  enhance  conventional  classroom  instruction  (e.g.,  providing  students  in  a 
biology  course  with  an  interactive  map  of  the  brain  to  help  them  leam  brain  functions).  Further 
along  the  continuum  are  courses  that  are  partly  delivered  over  the  Web,  such  as  elective  modules 
that  supplement  classroom  instruction.  Clearly,  factors  beyond  pedagogy  such  as  technical 
reliability,  interface  design,  and  evaluation  become  more  important  at  this  level.  Finally,  at  the 
“macro”  end  of  the  continuum  are  complete  distance  learning  programs  and  virtual  universities. 

Other  researchers  have  recognized  the  importance  of  determining  the  level  of  Web-use  in 
a  course.  For  example,  Galloway  (1998)  identified  three  levels  of  Web-use.  In  Level  1,  the  Web 
is  used  to  post  course  material  with  little  or  no  online  instruction.  The  instructor  guides  students 
to  the  relevant  information  rather  than  obliging  the  students  to  search  for  information.  In  Level 
2,  the  Web  is  used  as  the  medium  of  instruction.  Course  assignments,  group  projects,  and  lecture 
notes  are  posted  on  the  Web.  The  teacher  becomes  the  facilitator  of  knowledge,  guiding  the 
student  rather  than  telling  them  what  to  do.  In  addition,  there  is  increased  student-student 
interaction.  Courses  that  are  offered  completely  online  fall  into  Level  3.  Teachers  and  students 
interact  only  over  the  Internet,  and  knowledge  of  using  the  technology  is  extremely  important  at 
this  level. 

Web-based  instruction  is  still  in  an  early  stage  of  implementation.  Nevertheless, 
educational  institutions,  private  industry,  the  government,  and  the  military  anticipate  immense 
growth  in  its  use.  In  its  1999  distance  education  statistical  analysis,  for  example,  the  National 
Center  for  Education  Statistics  found  that  82%  of  higher  education  institutions  plan  to  increase 
their  use  of  Web-delivered  courses  (Johnson,  2001).  In  addition,  it  is  estimated  that  by  2003, 
approximately  $1 1.4  billion  will  be  spent  for  online  training  and  more  than  30%  of  corporate 
education  will  be  conducted  online  (Bonk  &  Cummings,  1998). 

Obstacles  to  realizing  the  Web’s  full  potential  for  learning  clearly  remain.  These  include 
the  appropriateness  of  pedagogical  practices  (Fisher,  2000)  and  the  bandwidth  bottleneck  for 
certain  learner  requests,  for  example  video  on  demand  (Saba,  2000).  From  an  evaluation 
perspective,  there  has  been  an  inclination  for  media  comparison,  such  as  the  effectiveness  of  a 
technology  relative  to  the  conventional  classroom  (Wisher  &  Champagne,  2000).  However,  the 
more  appropriate  benchmark  might  be  a  comparison  to  the  historical  findings  on  the 
effectiveness  of  conventional  CBI.  An  assessment  of  current  practices,  then,  must  consider 
whether  the  capabilities  of  the  Web  are  being  tapped,  how  interpretable  the  findings  are,  and  how 
strong  those  findings  are  in  comparison  to  conventional  CBI. 

The  Effectiveness  of  CBI 

CBI  has  been  a  significant  part  of  educational  technology,  beginning  with  the  first 
reported  use  of  the  computer  instructional  purposes  in  1957  (Saettler,  1990).  Its  emergence  as  a 
true  multimedia  delivery  device  occurred  in  the  early  1980s  with  the  coupling  of  a  videodisc 
player  with  a  computer.  In  recent  years,  the  videodisc  has  been  replaced  by  the  CD-ROM.  The 
combination  of  a  computer  controlling  quality  video  and/or  audio  segments  was  a  compelling 
advancement  in  CBI,  and  the  instructional  effectiveness  of  this  pairing  has  been  documented. 
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Since  our  premise  is  that  Web-based  instruction  should  be  at  least  “as  good  as”  conventional 
CBI,  a  summary  of  the  findings  is  needed  in  order  to  identify  a  comparative  benchmark. 

Fletcher  (1990)  conducted  a  quantitative  analysis  of  the  education  and  training 
effectiveness  of  interactive  videodisc  instruction.  Specifically,  empirical  studies  comparing 
interactive  videodisc  instruction  to  conventional  instruction  were  segmented  into  three  groups: 
higher  education,  industrial  training,  and  military  training.  The  various  learning  outcomes 
investigated  included:  (1)  Knowledge  outcomes,  which  assessed  a  student’s  knowledge  of  facts 
or  concepts  presented  in  the  instructional  program;  (2)  performance  outcomes,  which  assessed  a 
student’s  skill  in  performing  a  task  or  procedure;  (3)  retention,  which  measured  the  durability  of 
learning  after  an  interval  of  no  instruction;  and  (4)  the  time  to  complete  the  instruction.  The 
effect  sizes,  or  the  difference  between  the  mean  scores  of  the  treatment  and  comparison  groups 
divided  by  the  standard  deviation  of  the  control  group,  were  computed  for  each  of  the  28  studies 
identified. 

The  results  of  the  Fletcher  (1990)  meta-analysis  are  presented  in  Table  1,  broken  down 
by  learning  outcome,  and  in  Table  2,  broken  down  by  instructional  group.  The  effect  sizes  are 
reported  in  standard  units  with  a  corresponding  percentile  from  a  normal  distribution. 

Table  1 

Average  Effect  Sizes  for  Four  Types  of  Knowledge  Outcomes  for  CBI 


Table  2 

Average  Effect  Sizes  for  Three  Instructional  Groups  Using  CBI 
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The  conclusion  of  the  Fletcher  (1990)  analysis  was  that  interactive  video  instruction  was  both 
effective  and  less  costly  than  conventional  instruction. 

In  a  later  analysis  of  the  effectiveness  of  CBI,  Kulik  (1994)  took  into  account  the 
conceptual  and  procedural  differences  in  how  the  computer  was  used  in  the  individual  studies. 

In  his  analysis  of  97  studies  that  compared  classes  that  used  CBI  to  classes  that  did  not,  Kulik 
(1994)  computed  the  overall  effect  size  as  well  the  effect  sizes  corresponding  to  five  categories 
of  computer  use  relevant  to  the  present  report:  1)  tutoring,  2)  managing,  3)  simulation,  4) 
enrichment,  and  5)  programming. 

Kulik  determined  the  overall  effect  size  to  be  .32.  This  indicates  that  the  average  student 
receiving  CBI  performed  better  than  the  average  student  in  a  conventional  class,  moving  from 
the  50th  percentile  to  the  62nd  percentile.  However,  the  effect  sizes  when  categorized  by 
computer-use  yielded  somewhat  discrepant  results.  These  results  are  presented  in  Table  3. 

Table  3 

Average  Effect  Sizes  for  Six  Categories  of  CBI 


Application 

Effect  Size 

Percentile 

Tutoring 

.38 

65th 

Managing 

.14 

56th 

Simulation 

.10 

54th 

Enrichment 

.14 

56th 

Programming 

.09 

53rd 

The  effect  size  for  computer-based  programs  used  for  tutoring  (.38)  is  significantly 
higher  than  the  rest,  indicating  that  students  who  use  computers  for  these  purposes  may  achieve 
better  outcomes  than  students  who  use  CBI  for  management,  simulation,  enrichment,  or 
programming  purposes.  In  addition,  it  is  clear  from  the  table  that  basic  programming  and 
simulations  had  minimal  effect  on  student  performance.  The  conclusion  of  the  Kulik  (1994) 
analysis  was  that  researchers  must  take  into  account  all  types  of  CBI  when  trying  to  assess  their 
effects  on  student  learning. 

Finally,  Liao  (1999)  conducted  a  meta-analysis  of  46  studies  that  compared  the  effects  on 
learning  of  hypermedia  instruction  (e.g.,  networks  of  related  text,  graphics,  audio,  and  video)  to 
different  types  of  non-hypermedia  instruction  (e.g.,  CBI,  text,  conventional,  videotape).  Results 
indicated  that,  overall,  the  use  of  hypermedia  in  instruction  results  in  more  positive  effects  on 
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students  learning  than  non-hypermedia  instruction  with  an  average  effect  size  equal  to  0.41. 
However,  the  effect  sizes  varied  greatly  across  studies  and  were  influenced  by  a  number  of 
characteristics.  Effect  sizes  were  larger  for  those  studies  that  used  a  one-group  repeated  measure 
design  and  simulation.  In  addition,  effect  sizes  were  larger  for  studies  that  compared  hypermedia 
instruction  to  videotaped  instruction  than  for  studies  that  compared  hypermedia  instruction  to 
CBI.  These  results  seem  to  suggest  that  the  results  of  hypermedia  instruction  greatly  depend  on 
the  type  on  instruction  that  it  is  compared  to. 

While  each  of  the  studies  reviewed  above  provides  evidence  for  the  positive  effects  of 
CBI  on  student  learning,  they  also  address  the  complexity  of  the  issue.  The  relationship  between 
CBI  and  learning  is  influenced  by  many  variables  including  the  type  of  media  being  used,  what 
CBI  is  being  compared  to,  and  the  type  of  research  design  employed,  to  name  just  a  few.  These 
issues  increase  in  complexity  when  applied  to  Web-based  instruction  as  instruction  becomes  less 
linear  and  more  interactive  and  dynamic. 

The  Effectiveness  of  Web-Based  Instruction 

The  review  encompassed  40  articles  published  between  1996  and  2002.  A  tabulation  of  the 
documented  findings  into  nine  dimensions  are  offered  in  Appendix  A,  along  with  our 
assessments  of  the  experimental  designs,  effect  sizes,  and  the  degree  to  which  the  evaluation 
incorporated  features  unique  to  Web-based  instruction.  The  studies  are  first  discussed  in  terms 
of  four  key  features:  1)  the  degree  of  interaction  in  the  course;  2)  the  measurement  of  learning 
outcomes;  3)  the  experimental  design  used  in  evaluating  the  course;  and  4)  the  extent  of  Web  use 
throughout  the  course.  Since  the  purpose  of  this  review  is  to  assess  current  practices  in  the  use 
and  evaluation  of  Web-based  courses,  the  criteria  that  guided  the  choice  of  studies  for  this  review 
were  broadly  defined.  The  studies  had  to  involve  the  use  of  the  Web  as  an  instructional  tool 
either  as  a  supplement  to  conventional  classroom  instruction  or  as  the  primary  medium  of 
instruction.  In  addition,  the  studies  had  to  include  an  evaluation  of  the  Web-based  components 
of  the  course.  The  studies  were  directed  at  the  undergraduate  and  graduate  levels  of  education. 

The  evaluations  include  in  these  studies  fell  into  two  broad  categories.  The  first  category 
of  evaluation  consisted  of  comparisons  of  Web-based  instructional  approaches  to  conventional 
classroom  instruction.  These  evaluations  could  involve  the  comparison  of  a  control  group  with 
an  experimental  group  derived  from  the  same  population  of  students  or  a  simple  comparison 
between  a  class  taught  at  one  time  without  the  use  of  the  Web  and  the  same  class  taught  at  a 
different  time  using  the  Web.  The  second  category  of  evaluations  involved  the  assessment  of 
student  performance  and  reactions  relative  to  a  single  course. 

Methods  and  Procedure 

The  Educational  Resources  Information  Center  (ERIC)  and  Psychological  Abstracts 
databases  were  searched  using  the  following  combinations  of  key  words:  “web-based  courses,” 
“web-based  instruction,”  “web-based  courses  and  evaluation,”  “course  evaluation  and  web,” 
“course  evaluation  and  Internet,”  “web  and  distance  education,”  and  “online  course  and 
evaluation.”  Because  we  were  trying  to  assess  current  practices  in  evaluating  Web-based 
instruction,  the  search  was  restricted  to  the  years  between  1996  and  2002.  This  search, 
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conducted  between  August  2000  and  July  2002,  identified  more  than  500  qualifying  studies. 
However,  most  of  these  studies  concerned  recommendations  for  the  design  of  online  courses  or 
technology  concerns  rather  than  an  evaluation  of  a  specific  course,  so  they  were  not  included  in 
the  review.  In  addition,  we  found  several  relevant  studies  from  the  Journal  of  Asynchronous 
Learning,  Education  at  a  Distance,  and  the  previous  four  years  of  the  Proceedings  of  the 
Distance  Teaching  and  Learning  Conference.  Finally,  we  received  useful  references  and 
citations  from  researchers  in  the  field  of  distance  education  that  had  not  been  identified  by  our 
search  of  the  databases. 

We  narrowed  the  larger  set  of  studies  to  47  based  on  reports  of  empirical  evidence  for 
instructional  effectiveness.  These  are  summarized  in  Appendix  A,  which  is  organized 
alphabetically  by  the  first  author’s  last  name.  Although  most  of  the  studies  involved  courses  in 
the  physical  sciences,  the  instructional  content  of  the  courses  concerned  a  variety  of  subject 
matters,  including  philosophy,  nutrition,  economics,  and  sports  science. 

Our  review  of  the  literature  on  Web-based  instruction  is  organized  according  to  three 
categories:  1)  study  characteristics,  2)  methodological  characteristics,  and  3)  course 
characteristics.  The  analysis  of  these  characteristics  provides  some  insight  into  what  questions 
people  are  asking  about  Web-based  instruction  and  how  well  they  are  being  answered. 
Descriptive  statistics  for  these  characteristics  are  presented  in  Table  4. 
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Table  4 

Descriptive  Statistics  for  Studies  Included  in  Review 


Variables 

■ 

I 

Variables 

N 

% 

Content  Area 

Course  Characteristics 

Math/Engineering/Computers 

ii 

23 

Level  of  Web  Use 

Science/Medicine 

12 

26 

All-Online 

29 

62 

Distance  Education  Programs 

7 

15 

Blend 

17 

36 

Social  Sciences/Education 

11 

23 

Mixed  between  courses 

1 

2 

Business 

4 

9 

Language 

2 

4 

Attrition  Data 

Yes 

18 

38 

Educational  Level 

No 

29 

62 

Undergraduate 

38 

81 

Graduate 

8 

17 

Variables  Assessed 

Both 

1 

2 

Course  design 

21 

45 

Demographics 

22 

47 

Methodological  Characteristics 

Computer  Experience 

13 

28 

Sample  Size 

Effectiveness  of  instructor 

18 

38 

9-50 

13 

28 

Technical  issues 

12 

26 

51-100 

10 

21 

Interaction/Participation 

12 

26 

Over  100 

13 

28 

Desire  to  take  additional  online 

6 

13 

Not  Reported 

11 

23 

Recommendation  of  course  to 

3 

6 

others 

Comparison  Group 

Yes 

29 

62 

No 

18 

38 

Study  Characteristics 


Content  Area 

Content  areas  represented  in  the  studies  were  wide-ranging.  Approximately  23% 
examined  the  effects  of  Web-based  instruction  for  teaching  math,  engineering,  and  computer 
courses  whereas  26%  focused  on  the  teaching  of  science  and  medical  courses.  Another  23% 
focused  on  the  social  sciences  and  education.  In  addition,  about  15%  of  the  studies  evaluated 
entire  programs  of  distance  learning,  which  most  likely  were  comprised  of  many  types  of 
courses.  The  wide  variety  of  content  areas  discovered  in  this  review  demonstrates  the  flexibility 
of  Web-based  instruction  to  be  adapted  to  the  requirements  of  students  and  teachers  in  many 
different  subject  areas. 

Educational  Level 

Both  undergraduate  and  graduate  students  were  represented  in  this  review.  Of  the  47 
studies,  81%  evaluated  Web-based  instruction  for  undergraduate  students,  17%  evaluated 
graduate  students,  and  2%  evaluated  Web-based  instruction  combined  for  both  graduate  and 
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undergraduate  students.  Given  the  differences  in  both  course  content  and  teaching  styles 
between  undergraduate  classes  and  graduate  classes,  it  would  have  been  interesting  to  assess  the 
differential  impact  of  Web-based  instruction  on  student  learning  between  the  two.  However, 
insufficient  information  was  provided  by  the  studies  to  be  able  to  draw  any  firm  conclusions  on 
this  issue.  For  example,  many  of  the  studies  failed  to  provide  means  or  standard  deviations  for 
learning  outcome  measures.  In  addition,  may  of  the  studies  with  graduate  students  did  not 
describe  the  methods  used  to  assess  student  learning  and  performance  or  provide  adequate 
descriptive  information  as  to  assess  course  content  and  level  of  Web  use. 


Methodological  Characteristics 

Sample  Size 

The  sample  size  of  a  study  can  significantly  affect  the  statistical  power  of  the  underlying 
tests  for  differences.  Of  the  47  studies,  36  reported  information  about  the  sample  sizes  of 
participants.  These  ranged  from  9  to  1 ,406  (M  =  1 53,  SD  =  245).  Of  these  36  studies  that 
provided  sample  sizes,  most  (64%)  had  sample  sizes  of  fewer  than  100  participants.  Effect  sizes 
were  available  for  10  of  these  studies,  and  the  mean  effect  size  was  approximately  0.09.  For 
studies  in  which  the  sample  size  exceeded  100,  the  mean  effect  size  increased  to  0.55.  In 
general,  the  larger  the  sample-size,  the  stronger  the  statistical  power.  However,  since  effect  sizes 
were  available  for  only  15  of  the  studies,  these  results  must  be  interpreted  with  caution. 

Use  of  A  Comparison  Group 

The  majority  of  the  studies  identified  for  this  review  used  a  comparison  group  in  which 
students  took  the  same  course  face-to-face  or  the  same  course  with  no  Web-based  components. 
However,  41%  of  the  studies  simply  evaluated  the  Web-based  course  with  no  comparison  group. 
A  similar  pattern  was  found  in  evaluations  of  distance  learning  technology  in  training 
environments  in  which  55%  of  evaluations  did  not  use  a  comparison  group  (Wisher  & 
Champagne,  2000).  While  a  comparison  group  is  not  a  requirement  for  course  evaluation,  the 
absence  of  one  can  threaten  the  internal  validity  of  the  study  and  restrict  the  interpretation  of  the 
data.  It  is  difficult  to  draw  conclusions  about  the  impact  of  instruction  using  the  Web  on  student 
learning,  satisfaction,  and  other  outcomes  without  an  equivalent  comparison  group. 

Furthermore,  of  the  29  studies  that  had  a  comparison  group,  only  one  (Schutte,  1996)  randomly 
assigned  students  to  conditions.  Thus,  even  most  of  the  studies  with  comparison  groups  were 
subject  to  many  possible  confounding  variables  that  may  have  influenced  the  relationships 
between  Web-based  instruction  and  learning  outcomes. 

Course  Characteristics 


Level  of  Web-Use 

As  demonstrated  by  the  results  of  both  Kulik  (1994)  and  Liao  (1999),  different  forms  of 
CBI  can  differentially  affect  student  outcomes.  Thus,  it  is  important  to  take  into  account  how  a 
particular  medium  of  instruction  is  applied  when  evaluating  a  course.  This  is  especially  true  of 
Web-based  instruction  in  view  of  the  medium’s  tremendous  ability  to  distribute  seemingly 
unlimited  resources  and  information  to  anyone  at  anytime.  The  flexibility  of  the  Web  allows  it 
to  be  used  for  a  variety  of  purposes,  from  course  administration  and  management  to  complete 
course  delivery,  and  each  of  these  “levels  of  use”  works  towards  a  different  goal. 
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All  of  the  studies  surpassed  the  use  of  the  Web  for  purely  management  purposes.  Of  the 
47  studies,  17  evaluated  “blend  courses”  or  courses  that  are  a  mix  of  both  face-to-face  instruction 
and  Web-based  components  (e.g.,  posting  of  course  syllabus  and  lecture  notes,  online  tutorials 
and  graphics,  etc.).  While  the  use  of  the  Web  in  these  “blend  courses”  fulfilled  many 
management  functions,  students  needed  to  access  the  Web  regularly  to  be  a  productive  member 
of  the  class.  This  access,  in  turn,  caused  higher  levels  of  Web-use  as  defined  by  the  frameworks 
described  above.  The  remainder  of  the  studies  included  courses  that  were  completely  online.  At 
this  level  of  use,  there  is  little  or  no  face-to-face  interaction,  so  evaluating  the  degree  of  online 
interaction  between  students  and  instructors  or  other  students,  the  availability  of  feedback,  and 
technological  issues  becomes  critical.  Of  the  30  courses  that  were  completely  online,  10 
evaluated  student  and  teacher  interactions,  the  availability  of  instructor  feedback,  and 
technological  issues.  Since  these  variables  are  the  cornerstones  of  good  instructional  design, 
regardless  of  delivery  medium,  it  is  important  that  they  be  included  in  any  assessment  of  Web- 
based  instruction. 

In  terms  of  learning  outcomes,  effect  sizes  were  available  for  six  of  the  “blend”  courses 
and  nine  of  the  all-online  courses.  The  mean  effect  size  for  the  “blend”  Courses  was  0.48  while 
the  mean  effect  size  for  the  all-online  courses  was  0.08.  Although  this  is  not  a  statistically 
significant  difference  (Mann- Whitney  U  Test,  p  =  .14),  the  direction  of  the  difference  is  the  same 
as  Liaos’s  (1999)  analysis,  which  showed  that  the  mean  effect  size  for  courses  that  used 
hypermedia  as  a  supplement  to  conventional  instruction  was  0.18  standard  units  higher  than 
courses  that  replaced  conventional  instruction  with  hypermedia.  Taken  together,  the  results 
appear  to  suggest  that  Web-based  instruction  may  be  more  beneficial  for  student  learning  when 
used  in  conjunction  with  conventional  classroom  instruction  but  a  larger  sample  is  needed. 

Variables  Assessed 

Much  of  the  research  that  evaluates  Web-based  instruction,  and  distance  education  as  a 
whole,  lacks  a  guiding  theoretical  framework  (Saba,  2000;  Berge  &  Mrozowski,  2001). 
Furthermore,  there  is  no  consensus  as  to  what  variables  are  important  to  examine  when 
evaluating  Web-based  courses.  Determining  the  evaluation  elements  becomes  more  complicated 
for  online  courses  than  for  face-to-face  instruction  as  online  courses  incorporate  unique  elements 
such  as  flexibility,  a  wide  range  of  corresponding  resources  and  tools,  and  technological 
considerations  among  others.  As  a  result,  researchers  have  assessed  a  wide  range  of  variables. 
For  the  studies  included  in  this  analysis,  the  variables  assessed  can  be  grouped  into  eight  major 
categories: 


•  Demographics  (age,  gender,  race,) 

•  Previous  computer/Intemet 
experience 

•  Course  design 

•  Effectiveness  of  the  instructor 


Technical  issues 

Level  of  participation/collaboration 

■Recommendation  of  course  to 
others 

Desire  to  take  additional  online 
courses 
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Although  many  of  the  studies  assessed  the  design  of  the  Web-based  course  and  the  demographic 
characteristics  of  the  participants,  fewer  evaluated  the  quality  of  interaction  or  collaboration  in 
the  course,  the  effectiveness  of  the  instructor,  and  the  technology  itself. 

Attrition 

It  has  been  widely  recognized  that  the  attrition  of  students  is  a  greater  problem  for  online 
courses  than  classroom  courses  (Phipps  &  Merisotis,  1999;  Terry,  2001).  In  addition,  some 
research  has  shown  that  blended  courses  should  be  considered  separately  from  completely  online 
courses  when  assessing  student  attrition  as  blended  courses  have  lower  attrition  rates  (Bonk 
2001).  However,  only  14  (34%)  of  the  studies  reported  information  about  attrition.  This  is 
unfortunate  because  of  the  lost  opportunity  to  better  understand  the  characteristics  and  motives 
underlying  students’  decisions  to  drop  online  courses. 

Comparison  to  CBI 

When  compared  to  conventional  classroom  instruction,  the  learning  outcomes  from 
conventional  CBI  have  demonstrated  effect  sizes  significantly  above  the  “no  significant 
difference”  threshold  (Fletcher,  1990;  Kulik,  1994).  The  original  premise  stated  that  if  Web- 
based  instruction  was  employed  properly,  and  used  to  achieve  specific  learning  objectives,  it 
should  lead  to  effect  sizes  that  are  at  least  comparable  to  CBI.  Of  the  15  studies  in  this  analysis 
that  provided  sufficient  information  to  calculate  effect  sizes,  eight  (53%)  of  the  effect  sizes  were 
positive  and  favored  the  group  that  used  Web-based  instruction,  while  seven  (47%),  were 
negative  and  favored  the  group  that  did  not  use  Web-based  instruction.  The  effect  sizes  ranged 
from  -.40  to  1.60.  The  grand  mean  for  all  15  effect  sizes  was  0.24,  and  the  grand  median  was 
0.095.  The  standard  deviation  of  0.58  indicates  the  great  variability  of  effect  sizes  across  studies. 

Because  these  data  did  not  meet  the  assumptions  of  normality  and  homoscedasticity,  non- 
parametric  tests  were  performed  on  the  mean  effect  sizes  of  the  current  analysis  as  well  as  the 
analyses  conducted  by  Kulik  (1994)  and  Liao  (1999)  to  compare  Web-based  instruction  to  CBI. 
Results  of  a  Kruskal-Wallis  Test  indicated  that  there  was  no  significant  difference  (p  =  .47)  in 
mean  effect  size  across  the  three  sets  of  analyses.  The  average  effect  sizes  and  their 
corresponding  standard  deviations  for  the  studies  included  in  this  article  and  the  previous 
analyses  conducted  by  Kulik  and  Liao  are  listed  in  Table  5. 

Table  5 
Comparison 
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These  results  would  seem  to  suggest  that  Web-based  instruction  is  “as  good”  as  CBI  but,  as  will 
be  discussed  in  the  following  section,  this  may  be  a  premature  conclusion. 

Conclusion 

On  the  basis  of  a  limited  number  of  empirical  studies,  Web-based  instruction  shows  an 
improvement  over  conventional  classroom  instruction.  However,  it  is  debatable  whether  Web- 
based  instruction  compares  favorably  to  CBI.  On  the  surface,  the  overall  effect  size  is  smaller, 
but  not  statistically  significant.  This,  of  course,  cannot  be  interpreted  to  say  that  they  are 
equivalent,  but  rather  that  thefe  is  no  detectable  difference.  This  is  apparently  due  to  inconsistent 
and  widespread  variability  in  the  findings.  As  the  number  of  studies  that  report  comparative  data 
increase,  leading  possibly  to  a  more  stable  central  tendency  in  the  effect  size,  a  more  reliable 
assessment  of  how  well  Web-based  instruction  compares  to  CBI  will  be  possible. 

There  are  numerous  reasons  why  the  effectiveness  of  Web-based  instruction  may  not  yet 
be  fully  realized.  For  example,  many  of  the  early  adapters  were  apparently  faculty  from  a 
diversity  of  fields  who  were  not  necessarily  trained  in  the  art  and  science  of  instructional  design. 
Their  comparisons  were,  in  some  cases,  a  first  attempt  at  Web-based  instruction  compared  to  a 
highly  practiced  classroom  rendition  of  the  course.  Another  restriction  may  have  been  response 
delays  not  uncommon  during  peak  usage  periods  on  the  Internet,  in  contrast  to  the  immediate 
responses  possible  with  a  stand-alone  computer:  With  packet-based  networks,  variable  delays 
cause  latency  problems  in  the  receipt  of  learning  material,  particularly  with  graphic  images. 
Previous  research  has  demonstrated  a  slight  decrement  in  learning  due  to  inherent  delays  of 
transmitting  complex  graphics  over  the  Internet  (Wisher  &  Cumow,  1999). 

One  objective  of  this  article  was  to  discuss  the  various  roles  that  the  Web  plays  in 
educational  courses  and  the  importance  of  identifying  this  factor  when  evaluating  courses.  Here, 
we  have  limited  our  analysis  to  those  applications  involving  direct  instruction  through  the  Web. 
As  described  earlier,  there  are  many  other  advantages  that  the  Web  offers  (e.g.  access,  flexibility, 
enrollment,  and  management)  that  must  be  factored  in  when  determining  the  overall  value  that 
the  Web  offers  to  a  learning  enterprise. 

How  large  a  learning  effect,  in  terms  of  an  effect  size,  can  we  expect  from  the  Web?  One 
possibility  comes  from  research  on  intelligent  tutoring  systems.  These  are  knowledge-based 
tutors  that  generate  customized  problems,  hints,  and  aids  for  the  individual  learner  as  opposed  to 
ad-hoc,  frame-oriented  instruction.  When  compared  to  classroom  instruction,  evaluations 
indicate  an  effect  size  of  1 .0  and  higher  (Woolf  &  Regian,  2000;  Wisher,  Macpherson. 
Abramson,  Thornton,  &  Dees,  2001).  If  these  individual  learning  systems  are  further 
complemented  by  collaborative  learning  tools  and  online  mentoring,  effect  sizes  on  the  order  of 
2  standard  deviation  units,  as  suggested  by  (Bloom,  1984),  may  someday  be  possible. 

The  use  of  the  Web  for  instruction  is  at  an  early  stage  of  development.  Until  now,  there 
has  been  a  lack  of  tools  for  instructional  developers  to  use,  but  this  is  beginning  to  change  with 
the  emergence  of  common  international  standards  (Fletcher  &  Dodds,  2001).  Such  standards 
and  specifications,  such  as  the  Sharable  Content  Object  Reference  Model,  allow  vendors  to 
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develop  educational  and  training  content,  authoring  tools,  and  repositories  that  are  interoperable. 
The  potential  of  Web-based  instruction  for  the  Army  will  be  furthered  as  pedagogical  practices 
improve,  advances  in  standards  for  structured  learning  content  progress,  and  improvements  in 
bandwidth  are  made.  For  now,  it  appears  that  Web-  based  instruction  is  moving  towards  the 
level  of  effectiveness  previously  achieved  by  CBI.  As  suggested  by  Fletcher  (2001),  the  Rule  of 
Thirds,  reduce  the  cost  of  instruction  by  about  one-third,  and  additionally  either  reduce  the  time 
of  instruction  by  about  one-third  or  increase  the  effectiveness  of  instruction  by  one-third,  is 
likely  to  apply  to  Web-based  instruction. 
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Appendix  A 

Characteristics  of  Studies  On  Web-Based  Instructional  Effectiveness 


Angulo  &  Bruce  (1999) 

Arvan  et  at.  (1998a) 
Arvan  et  al.  (1998b) 


Arvan  et  al.  (1998c) 
Arvan  et  al.  (1998d) 


Arvan  et  al.  (1998e 


Arvan  et  al.  (1998 
Bee&Usip  (1998) 


Cooper  (2001) 
Davies  &  Mendenhall 


Fredericksen  et  al. 
(2000a) 


Fredericksen  et  al. 
(2000b) 


Course  Evaluated 


Psychology,  Drama. 
Philosophy,  English 
Literature.  Chemistry 


Electrical  and  Computer 

_ Engineering _ 

Chemis 


Educational 

Level 


Undergraduate 
and  Graduate 

Undergraduate 

Undergraduate 


Undergraduate 


Comparison  Attrition 


118  (78  Web, 
40  No  Web) 
304  (203  Web, 
101  No  Web 


Fundamentals  of  Computer 

_ Applications _ 

Fitness  and  Lifestyle 

_ Management _ 

Program-wide  evaluation  of 
SUNY  Learning  Network 
(online  instructional 

_ program) _ 

Program-wide  evaluation  of 
the  Internet  Academy  at 
Herkimer  County  Community 


Undergraduate 


Undergraduate 

Undergraduate 


153  (78  Web, 
75  No  Web) 


131  (37  Web, 
94  No  Web) 

'  96  (57  Web.  39 
No  Web) 


All  Online  Not  Reported  Yes 


Johnson  (2001) 

Political  Science 

Undergraduate 

All  Online 

Johnson,  Aragon,  Shalk, 
&  Palma-Rivas  (2000) 

Human  Resources 

Graduate 

All  Online 

Jones (1999) 

Statistics 

Undergraduate 

All  Online 

LaRose,  Gregg,  &  Eastin  I 


Leasure,  Davis,  & 
Thievon  (2000) 

Magalhaes  &  Schiel 


Telecommunications 
Nursing  Research 
Graphic  Mechanics 


Maki  et  al.  (2000) 

Psychology 

Undergraduate 

Blend 

Maki  &  Maki  (2002) 

Psychology 

Undergraduate 

Blend 

McNulty  et  al.  (2000) 

Anatomy 

Undergraduate 

Blend 

Murphy  (2000) 

General  Solis 

Undergraduate 

Blend 

Navarro  &  Shoemaker 

(2000) _ 

Phelps  &  Reynolds 


Powers.  Davis,  & 
Torrence  (1998) 


Meteorology 


Instructional  Technology, 
Information  Technology  and 
Media  Literacy, 
Technologies  of  Distance 


Undergraduate 

Undergraduate 


AH  QnNne  151'NoWebj 
All  Online  Not  Reported 


Ryan,  Carlton,  &  Ali 
(1999) 

7  Nursing  Courses 

Graduate 

Sandercock  &  Shaw 

Sports  Science 

Undergraduate 

N/A 

N/A 

Exam 

Scores 

-0.19 

Exam 

Scores 

0.36 

N/A 

N/A 

Schlough  &  Bhuripanyo  Task  Analysis 

(1998) _ _ _ _ 

Organizational  Behavior, 
Personal  Finance, 

^  ^  Managerial  Accounting, 

Schulman  &  Sims  (1999)  Soci0|ogjca|  Foundations  of 

Education,  Environmental 

_ Studies _ 

Schutte  (1 996)  Social  Statistics 


Undergraduate 

Undergraduate 

Undergraduate 


All  Online  22  No 


„  99  (40  Web,  59 

All  Online  No  Web)  Yes 


No 

N/A 

N/A 

Yes 

N/A 

N/A 

No 

Exam 

Scores 

-0.13 

No 

N/A 

N/A 

Yes 

N/A 

N/A 

|33  (16  Web,  17 


No 

N/A 

N/A 

No 

N/A 

N/A 

No 

Assignment 

Scores; 

Exam 

Scores 

1.11;  0.095 

No 

N/A 

N/A 

No 

Exam 

Scores 

0.012 

Yes 

